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ABSTRACT 

A large population of z > 5 Lyman break galaxies has been identified in recent years. 
However, the high redshift galaxies selected by difi^erent surveys are subject to a va- 
riety of selection effects - some overt, others more subtle. We present an analysis of 
sample completeness and contamination issues in high redshift surveys, focusing on 
surveys at z « 5 and using a spectroscopically-confirmed low redshift sample from 
the DEEP2 survey in order to characterise contaminant galaxies. We find that most 
surveys underestimate their contamination from highly clustered galaxies at z « 1 
and stars. We consider the consequences of this for both the rest-frame ultraviolet 
luminosity function and the clustering signal from z « 5 galaxies. We also find that 
sources with moderate strength Lyman-a emission lines can be omitted from dropout 
surveys due to their blue colours, again effecting the derived luminosity functions. We 
discuss the points of comparison between different samples, and the applicability of 
survey-specific results to the population at z > 5 in general. 

Key words: galaxies: high-redshift, evolution, luminosity function; techniques: pho- 
tometric 



1 INTRODUCTION 

] The study of 2: > 5 galaxies has become a well-developed 
■ field in recent years. The Lyman Break (or 'dropout') tech- 
' nique - which identifies starbursting sources by the dramatic 
, spectral break around the rest -frame Lyman-alph a feature 
and was first applied at z ~ 3 jSteidel et al.|[l999h - is now 
widely applied on large multicolour datasets. As a result, 
large samples of photometrically-selected high-redshift can- 
didates can be constructed with a comparatively small in- 
vestment of telescope time. Such photometric samples have 
been widely used to derive statistical properties for z > 5 
galaxies as a whole, i ncluding luminosity functions and clus- 
tering pa.ramet ers. l|Quchi et al.l l2004bl : llwata et ahl l2007l : 
iLee et al.|[200^ ) 

However, the high redshift galaxies selected by differ- 
ent surveys are subject to a variety of selection effects - 
some overt, others more subtle. While samples of Lyman- 
a emitting galaxies (selected over a narrow redshift range, 
and relatively easy to confirm spectroscopically due to the 
presence of strong emissio n lines) have compara tively sim- 
ple selection windows (see lHaves fc 6stlinll2006l . for discus- 
sion), the variety in width and shape of broadband filters 
leads to a wide range of redshifts (and hence intrinsic lu- 
minosity limits) falling into the selection window, while the 
presence or absence of emission lines can alter the effec- 
tiveness of a colour selection. Such effects are highly sen- 



sitive to filter profile, selection criteria and survey depth. 
Although attempts to interpret the results of photometric 
surveys have generally considered each such survey compa- 
rable, these sample-specific issues relating to completeness 
and contamination can alter the resulting physical interpre- 
tations. The situation is complicated by the fact that the 
dominant contaminant populations have not hitherto been 
carefully and systematically characterised. 

While most surveys take into account a subset of these 
effects, the description of published data is in many cases 
inadequate to allow comparison between results. In this pa- 
per, we present an analysis of significant sample complete- 
ness and contamination issues in high redshift surveys based 
on the Lyman break technique, focusing on the galaxy pop- 
ulation at 2: « 5 to illustrate the complexity of the situation. 
We discuss the points of comparison between different sam- 
ples, and the applicability of survey-specific results to the 
population at 2; > 5 in general, pulling together the mod- 
elling and observational results to construct a coherent pic- 
ture of the high redshift population under examination. An 
understanding of these issues is essential to allow fair com- 
parison between different datasets, between data and theory 
and when planning future surveys. 

In section [5] we consider the selection effects that can 
influence a high redshift survey and in section |3] we describe 
the contaminant populations that also show dropout colours. 
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In section |4] we discuss the consequences of these effects on 
the interpretation of pubhshed results from high redshift 
surveys, and the imphcations for the design of future sur- 
veys. 

Throughout the paper, we consider filter sets and se- 
lection functions that have been applied to existing 2; « 5 
Lyman break galaxy surveys. The instruments and filter sets 
considered, together with examples of relevant surveys are 
shown in table [1] and illustrated in figures [T] (filter profiles) 
and [2] (colour selections). While these selection functions do 
not form constitute an exhaustive list, they are indicative 
of the variation from survey to survejQ. Where appropri- 
ate, we adopt the following cosmology: a flat Universe with 
Q.A = 0.7, Q.M = 0.3 and Ho = 70/i7okms"^ Mpc"^ AU 
mag nitudes (optical and infrared) are quoted in the AB sys- 
tem (jOke fc Gunnlll983h . 



2 COMPLETENESS ISSUES IN DROPOUT 
SAMPLES 

2.1 Redshift Completeness and Number Counts 

The majority of z > 5 surveys published in the literature 
fall into two categories. Sources are selected either for the 
presence of a strong emission line, detected as excess emis- 
sion in narrowband im aging(not discussed further here, see 
iHaves fc Ostlinll2006l ). or by the break in their continuum 
emission. In the latter case, the selection is based on an 
extreme colour in a single pair of filters, often refined by 
constraints in one or more additional bands. In the case of 
galaxies at z « 5, the selection criteria applied in most sur- 
veys is either based on R — I colour or on V — /, depending 
on available imaging, with a constraint on I — Z often ap- 
plied a posteriori in order to reduce sample contamination 
(see section [31). 

The requirement for an extreme colour sets a firm lower 
limit to the redshift of a survey, while the width of the selec- 
tion filters defines the redshift range. In theory, the simplic- 
ity of this approach should allow samples selected using the 
same colour cuts on different instruments to be compared 
directly. 

However such comparisons overlook one significant fac- 
tor: not all filter sets are equivalent. There has been a prolif- 
eration of filter sets in the optical, each designed to optimise 
source signal to noise, but approaching the problem in differ- 
ent ways. Such filters are assigned common names of V, R, 
I and Z (or variants on these) on the basis of their effective 
wavelength, but often differ significantly in terms of width, 
filter respons60 and overlap with neighbouring filters. This 
has little effect on the measured magnitudes in each band 
of sources with smooth profiles, but can significantly impact 
the measured colours of spectra dominated by sharp features 
such as continuum breaks. 

^ Not e, we don't discuss the UKIDSS selection of iMcLure et all 
in detail since the R — Z > 3 colour cut excludes z < 
5.5 galaxies and renders this more properly an j'-drop selection. 
Hence the UKIDSS survey expects sources at much lower surface 
densities making it not comparable to other 2 5 surveys. 
^ combining filter transmission profile with available information 
on CCD and instrumental response. 



As figure [3] illustrates, a single high redshift galaxy (in 
this case modelled as a source flat in f^, appropriate for 
young starbursts, and modulated by the intergalactic hydro- 
gen opacity as a function of redshift given by Madau 19951 ). 
can have dramatically different colours in different fllter sets. 
A single colour cut simply cannot be applied uniformly to 
different filter sets since the resulting redshift ranges can 
differ by Az — 1, particularly if V-drops are compared with 
sources selected as i?-drops. 

This simple fact is widely understood in the observa- 
tional community and most surveys account for this effect 
by varying their selection criteria accordingly as shown in 
table [1] to tune the minimum redshift satisfying the colour 
selection. Nonetheless, the redshift range probed still varies 
from sample to sample due to the width of, and overlap 
between, the filters available to them. 

The most dramatic effect of this redshift variation from 
survey to survey arises not from the redshift evolution of 
the population itself, but rather from the simple change in 
luminosity distance over the redshift range in question. For a 
magnitude-limited sample, a progressively brighter limiting 
luminosity is reached in each successive redshift slice. Given 
the relatively steep faint-end luminosity function observed 
for Lyman break galaxies at z « 3, a small reduction in 
survey depth at any given redshift can have dramatic effects 
on the number of galaxies predicted in that bin. 

The combination of these effects - filter response, sub- 
sequent redshift selections and the luminosity bias towards 
low redshifts - and their effect on predicted number counts 
is illustrated in figure |31 For a constant luminosity function 
(i.e. a non-evolving population from 2: « 3) the number of 
galaxies predicted as a function of redshift is shown for each 
of the selection criteria in table [T] 

While the two V-dmp samples considered here (the se- 
lection functions generally applied to Subaru and HST/ ACS 
data) overlap signiflcantly in redshift with the ii-drop sam- 
ples, allowing both groups to be called 'z ~ 5 Lyman Break 
galaxy populations', the surface density of sources observed 
by such samples is some two times higher. Similarly, there 
are clear variations within the ii-drop samples. The 'Riz' se- 
lection utilised by the Subaru Deep Field (SDF) for example, 
will identify sources at lower redshifts than that applied by 
the ERGS survey at the VLT, but will miss 40% of Lyman 
break galaxies with the same continuum magnitude at z > 5. 

Also notable is the role played by photometric error in 
the redshift distribution. As figure|3]illustrated, it is possible 
for the dropout colour to remain static with redshift in some 
filter sets (due either to fllter overlaps or to signiflcant offsets 
between the R and I fllters). If the selection criteria is set 
close to such a plateau, then Gaussian scatter in both the 
intrinsic colour of the sources and the photometry will pro- 
mote a fraction of the more abundant population from lower 
redshifts into the sample, while simultaneously scattering a 
fraction of higher redshift sources blueward of the limiting 
colour. The result is a redshift distribution that bulges be- 
low the nominal selection cutoff, as seen in flgure|4]for both 
the VLT/F0RS2 and CFHT/MegaCam selection criteria. It 
is interesting to note that this effect will be present for any 
selection criteria based largely on a single colour, affecting 
Lyman-break galaxies across a range of redshif ts, as well as 
the D istant Red Galaxy population (DRGs, iFranx et al.l 
I2OO3II 
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Figure 1. The filter sets considered in this paper. In each case the published filter transmission function is convolved with the appropriate 
instrumental and CCD response to obtain the final curves. 
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Figure 2. The colour selection criteria considered in this paper. The colours expected of a fiat continuum source (i.e f\ oc A~^) at high 
redshift and a mature galaxy at intermediate redshift are shown for reference in solid and dotted lines respectively. The high redshift 
sources are flat in f^. The intermediate redshift interlopers are constructed from the population synthesis models of Maxai^ton (20o3)i 
assuming a formation redshift of 2 = 5 and a star formation timescale r =0.5 Gyr. Variation in star formation histories causes scatter in 
the low redshift galaxy locus. The colours shown in these plots are used consistently thro ughout the paper to represent e ach filter set. 
The da rk shaded region on the Subaru R plot shows the more liberal selection criteria of lYoshida et all 1 I2OO6I) relative to lOuchi et al.l 
Tnale shadinerV 
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Facility 



Filters Used 



Selection Criteria 



Examples 



HST/ACS 

VLT/FORS2 

Subaru/Suprime-Cam 

CFHT/Megacam 



F435W(f<), F606W(i>), 
F775W(i'), F850LP(2') 
B(ESO 74), i?(ESO 76), 
/(ESQ 77), Z(ESO 78) 
B, V, Rc, i', z' 



R 



i' > 1.3 
- / > 1.3 



iBremer et~all 1120041) : I Verma et al.l ||2007^ 



prep), 



V -i' > 1.2, i' ~ z' < 0.7 & 
V~i' > 1.8(j' - z') + 1-7 
Rc-i' > 1.2, i' - z' < 0.7 & 
Rc-i' > 1.0(i' - z') + 1.0 
r' - i' > 1.3 



Douglas et al (ir 

iLehnert &: Bremed ll2003l') 
'Vi2': l0uchi et al. r il2004ah .l Yoshida et al.l 

' Riz': Ouchi et al.l l^2004a^ . lYoshida et al.l 

iHooi) 

None as yet 



Table 1. The filter sets and selection criteria examined in this study. The last column gives examples of hi gh redshift studies using 
the selection criteria in question. Filter profiles are convolved with the CCD sensitivity at the given facility. iVanzella et al., (2006) used 
VLT/FORS2 for spectroscopy. T he VLT/FORS2 ERG S survey will be fully described in a forthcoming paper by Do uglas et al and has 
alread y yielded one z = 5.4 AGN llDouglas et al.ll2007l) and dozens of spectroscopically confi rmed sources a t z = 5 — 6. lLehnert fc Bremej 
l|2003l ') used a more restrictive R — I > 1.5 colour cut but the s ame VLT/FORS2 dataset. lYoshida et all l|2006t ) slightly modified their 
selection criteria to allow bluer objects than lOuchi et a~ I ll2004al') as shown in figure [2] 



Figur e [4] is plotted for th e luminosity function deter- 
mined by ISteidel et al] (|l999l ) for Lyman break galaxies 
a,t z — 3. This luminosity function is known to overpre- 
dict the number of z > 5 sources of bright magnitudes 
iLehnert fc Bremeill2003l ). and hence the predicted number 
counts in figure|4]should be considered indicative rather than 
precise. However, while the normalisation of figure |4] may 
change, the basic differences in the redshift distributions of 
sources will remain unless the shapes of those distributions 
also change. 

Altering the luminosity function parameters has no ef- 
fect on the colours of a high redshift source (which are fixed 
by rest-frame UV slope and Lyman-a forest transmission), 
but does effect the number of sources in a given volume 
which will satisfy this criterion. Decreasing the typical lumi- 
nosity L* to half its value at z — 3 reduces the peak number 
density of sources by a factor of six, but affects the shape less 
severely, broadening the redshift distribution at FWHM by 
approximately 10%. Similarly, increasing the faint end slope 
a from -1.5 to -1.9 has the effect of decreasing the peak num- 
ber density by 15% while leaving the shape and FWHM of 
the distribution unchanged. 

Hence the primary effect is on the normalisation rather 
than the shape of the redshift distribution and the filter-to- 
filter comparison discussed here are largely independent of 
luminosity function. 

The effect of changing the limiting luminosity of a sur- 
vey is rather more pronounced. Such a change does not alter 
the minimum redshift identified by the survey but does in- 
crease the depth relative to L* in each successive redshift 
bin. Since the luminosity function is steep at the faint end, 
this has the effect of broadening the redshift distribution in 
a given filter set and extending the tail of the function to 
higher redshifts. 

2.2 Effect of Line Emission 

Lyman break galaxy surveys focus on the analysis of galax- 
ies detected through their rest-frame ultraviolet continua. 
However, it is known from Lyman-Q emitter surveys (e.g . 
lAiiki et aLlliooel : IHu et ahllioM : iMalhotra fc RhoadsllioM ) 
that there exists a significant population of sources with 
powerful emission lines at Arcst ~ 1216A, a fraction of 
which overlaps with the Lyman break galaxy population. At 




^..^ Subaru R 

: CFHT 

0.0 !^ , , , \ , , , , \ , , , , \ , , , , 

4.0 4.5 5.0 5.5 6.0 

Redshift 

Figure 3. The effect of filter profile on redshift selection function. 
A uniform colour cut of "R-I or V-I>1.3" will select galaxies with 
minimum redshifts between z = 4.4 and z = 5.5 depending on the 
filter set in question. Also, due to filter overlaps, the colour can 
fiatten out as a function of redshift in some redshift ranges lead- 
ing to an increased effect from photometric scatter and intrinsic 
variation of spectral slope around the selection colour. 

z « 3, a quarter of all spectroscopically confirmed Lyman 
break galaxies show Lyman-a emission with a rest-frame 
equivalent widt h of Wo > 20A, whi le a second quartile has 
< W^o < 20A (|Shaplev et al.ll2003l ). 

Figure [5] illustrates the effect of moderate line emis- 
sion on the colours of a flat-spectrum continuum-source, for 
one example filter set - in this case, the VLT/F0RS2 fil- 
ters. As in most surveys, a fiat-spectrum galaxy will satisfy 
the VLT/FORS2 selection criterion at redshifts where the 
Lyman-Q feature is in the Jil-band. While this remains true, 
increasing line strength has the effect of reducing the galaxy 
colour and dropping it out of the sample at the low redshift 
end of the selection function. At higher redshift, the pres- 
ence of a hne-emitting population has the effect of scattering 
galaxies over a much broader region than the simple galaxy 
locus, potentially reducing the ability of a survey to reliably 
separate galaxy loci from contaminants (see section |3}. 
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Figure 4. The redshift distributions oi z > 4.5 sources, selected 
using different filter sets and the selection criteria of appropri- 
ate surveys. Both the surface density of sources and their red- 
shift ranges depend sensitively on the selection criteria and filter 
shapes. Counts are plotted a ssuming the z = 3 Lyman Break 
galaxy luminosity function of ISteidel et al.l lll999h . and as such 
overpredict the observed number counts at 2: ~ 5, and should be 
viewed as indicative rather than predictive. Luminosity function 
effects are discussed in the text. A typical scatter of ±0.15 mag- 
nitudes is assumed, due to scatter in both the intrinsic colours 
and the photometry at these faint limits. The input spectrum 
is a source fiat in . As discussed in section 12.21 and figure [E] 
changing the spectral slope (for example due to the presence of 
dust) has less effect on the redshift distribution than increasing 
the photometric scatter. 



A second effect of line emission is to increase the ob- 
served magnitude of a galaxy when compared to an equiv- 
alent continuum source without line flux. At faint magni- 
tudes, a moderate strength emission line can contribute a 
large fraction of the observed broadband flux, particularly at 
high redshifts (since only a small fraction of the filter is free 
of Lyman-a forest line dampening). Given the steep faint- 
end slope expected for any reasonable luminosity function, 
the population of galaxies just beyond the (continuum) mag- 
nitude limit of a survey is larger than the faintest population 
above the limit. Hence a small fraction of those galaxies, en- 
tering the sample due to the presence of strong emission 
lines, can skew both redshift and equivalent width distribu- 
tions; 

IStanwav et al.l l|2007h recently found evidence for a tail 
of high equivalent width sources in a faint Lyman break 
galaxy s ample at z ~ 6 from the Ifubble Ultra Deep Field 
(Bcckwi th et al.ll2006l ). suggesting that the effect of strong 
line emission on the tail of the galaxy distribution could be 
measurable in a larger sample at equivalent faint limits. 

Each of these effects will be dependent on the position of 
Lyman-a emission relative to the filter edges, and also on the 
detailed profile of those filters. A filter with a square-edged 
transmission profile will have clear advantages in terms of 
reducing the tail of galaxies that are detected in spectral re- 
gions with very little filter throughput. However the decline 
in throughput of optical CCDs through the Z-band leads to 




Figure 5. The effect of moderate-strength Lyman-o line emission 
on the R-I, I-Z colours of high redshift galaxies, illustrated in the 
ESO filter set used at the VLT. Rest-frame equivalent widths of 
zero to 95A are shown from lower-right to upper-left in increments 
of 5A, imposed upon a constant continuum fiux. At z < 5 the 
emission line is in the ij-band, while at 5 < z < 6 the emission 
line is in the /-band and hence affects both colours. Above z = 
6, the emission line enters the Z-band, before leaving the filter 
set entirely. Colours are for a source with constant Fu. If strong 
line emitters have blue spectral slopes, the effect would be more 
pronounced still (see figure |6ll. 

an inevitable blurring of the edges, and square-transmission 
filters will not prevent effects arising from incomplete blan- 
keting of one or more bands by Lyman-a forest absorption. 



2.3 Effect of Spectral Slope and Burst Age 

By contrast, intrinsic spectral slope has a relatively small 
effect on the colours of high redshift galaxies, which are 
dominated instead by the effects of interstellar absorption. 
Nonetheless, a blue rest-frame ultraviolet spectral slope 
(appropriate for young sources) might reasonably be as- 
sociated with strong line emission, and so will strengthen 
the effect seen above. The effect of varying the ultravi- 
olet spectral slope (in the absence of line emission) is 
shown in figure |6l The effects of a bluer average ultravi- 
olet continmun_^^ steep er than /3 = 2.0, 
see IStanwav. McMahon. fc Bunked l200a) compared with 
the lower redshift population (typically 13 = 1.1 — 1.5, 
IStcidcl et al. 1999) are unlikely to have a big impact on the 
selection since they are of the same order as photometric 
errors at these faint magnitudes, but could produce a mea- 
surable effect for sources close to the selection limits. 

The most significant factor driving the rest-UV spectral 
slope is the age of the current starburst in a galaxy. The 
hottest, most-massive stars provide the largest contribution 
to the rest-frame ultraviolet, and these stars also have the 
shortest lifetimes. As a result, the optical colours of galaxies 
at z « 5 evolve with the age of the galaxies in question. 
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Figure 6. The effect of varying tiie rest-frame ultraviolet slope, 
defined as fi, oc X~^, on tlie R-I, I-Z colours of liigli redshift 
galaxies, illustrated in the ESO filter set used at the VLT. A 
source with constant has a spectral slope /3 = 2.0. Sources 
with very blue spectral slopes will be lost close to the selection 
boundaries, although this effect is small compared with that due 
to emission lines (figure [Sjl. 
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Figure 7. The effect of stellar population age on the model 
colours of high redshift galaxies. Here we show the colours of 
galaxies forming stars continuously over 10 (dotted) and 100 Myr 
(dot-dash), contrasted with the fiat spectrum expected for a 
young instantaneous starburst (solid). The colours of the widely 
used Bruzual & Chariot (2003) models are shown in pale blue, 
while those predicted by the more recent Maraston (2005) mod- 
els are in purple. The Subaru i?-band selection filters and colour 
criteria are used. 



In figure [7| we explore the effect of changing the tem- 
plate SED on the selectability of high redshift galaxies, 
with particular reference to the Subaru i?-band colour se- 
lection in table [1] The colours of a spectrum fiat in fv is 
compared with those predicted by stellar population syn- 
thesis models for galaxies constant star formation over the 
preceding 10 Myr and 100 Myr (assuming solar metallicity 
and a Salpeter IMF). We consider the results of two dif- 
ferent stell ar synthesis codes for Mentical output param- 
eters. The iBruzual fc CharlotI ((200311 codes have been the 
most widel y used synth e sis mod els in recent years and were 
utilised by lOuchi et alj (|2004al ) to estimate the photomet- 
ric selection criteria for high redshift galaxies. They produce 
redder model colours at a given redshift than the newer stel- 
lar synthesis models of iMaraston ( 2005i ). which incorporate 
improved treatment of the thermally- pulsating asymptotic 
giant branch phase (see lBruzuaj|2007l ). 

Evidence from SED fitting to the stellar populations 
in jz > 5 galaxi es l|Verma et all l2007l : IStark et all l2007l : 
lEvles et al.l I2OO5I ) has determined that the majority of 
this population comprise either young starbursts (<50Myr) 
or young starbursts with an underlying older (>200Myr) 
population that no longer contributes significantly to the 
rest-UV. In either case, no compelling evidence has been 
found for continuous star formation over timescales of 
100 Myr. This youth of stellar po pulations at high redshift 
is supported by other work (sec Lchnort & Bremer '2003'; 
IStanwav. McMahon. fc Bunk er 2005; Pcntcricci ct al. 2007), 
which finds colours consistent with a flat spectrum in bands 
uncontaminated by the break in 2 > 4 samples. 

For the purposes of this paper we adopt a spectrum fiat 
in (i.e. (3 = 2.0) as being a simple template galaxy. As 



figure [7] illustrates, this is appropria te for young starburst 
populations (in the iMarastonI [2005 1 models), although we 
note that uncertainties in the star formation history can 
lead to variation in colour by approximately 0.1 mag (or less 
than the photometric scatter in a typical survey of faint 
dropouts). 

2.4 UV-faint Galaxies 

The dropout selection technique, by its very nature, 
identifies sources with a bright rest-frame ultraviolet 
continuum. Ultraviolet fiux is contributed primar- 
ily by young, short lived stars - hence the use of 
Lyman break selected samples to gauge the star for- 



mation hist or y of the universe ( Madau et al. _ 19961: 



Steidel et al.l 19991: Bouwens. Broadhurst. fc lUinerwortlil 



2OO3I : IStanwav. Bunker, fc McMahonll2003h 



However, the ultraviolet continuum flux declines 
sharpl y with increasing tim e after an instantaneous star- 
burst l)Leitherer et al.|[l999l ). varying by three orders of mag- 
nitude in the flrst 100 Myr. Hence the Lyman break tech- 
nique cannot identify galaxies at high redshift that have 
passed through a starburst and are now passively evolving 
since such sources will no longer have a strong ultraviolet 
continuum. At 2: 5 a 100 Myr-old starburst would have 
peaked at only z ~ 5.5 and yet may be undetectable by 
dropout selection (unless star formation continues through- 
out this time). 

Similarly, it takes a finite time after a starburst for an 
ultraviolet continuum fiux generating population to become 
established, causing very young starbursts (<5Myr) to be 
detectable primarily in emission from the Lyman-a emission 
line. Hence both very young and very old starbursts may 
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be omitted from a dropout survey, while galaxies of 10 — 
100 Myrs in age may be preferentially selected based on 
ultraviolet flux. 

Whenever only isolated subsets of the total galaxy pop- 
ulation is observed, it is possible to miss aspects of the bigger 
picture. Populations that are small in number - such as the 
DRG galaxies (a combination of old and dusty galaxies) at 
z ~ 2 - can make non-negligible contributions to important 
cosmological parameters such as the galaxy metal budget 
Bouche. Lehnert. fc Peroux|[200^ and stellar mass density 



Marchesini et al.l 20071 ). Hence it is important to consider 



constraints on galaxies not selected by high redshift dropout 
samples. 

An analogue to the passively evolving subset of z « 
2 DRGs at 2 ~ 5 would be old sources without signif- 
icant ongoing star formation. In these old galaxies, the 
most easily detecte d spectral feature wou ld be the rest- 
frame 4000A break. iMobasher et al.1 (|2005l ) have proposed 
that infrared data can be used to detect the Balmer break 
in 2 > 5 galaxies. Although their initial candidate may 
well lie at lower redshifts, the technique has now identi- 
fied a small number of candidate galaxie s at comparatively 
brigh t magnitudes (ICharv et al. 2007 : Rodighiero et al.l 



I2OO7I : iDunlop. Cirasuolo. fc McLurd l2007l l. Unfortunatelv 
a degeneracy in observed optical-infrared colour between 
2 ~ 2 dusty galaxies and non-starforming 2 > 4 galaxies 
renders the analysis of such a population based on photo- 
metric redshifts alone difBcult. Photometric redshifts in the 
available broad wavebands can favour one solution, but not 
to the exclusion of the other, and quantitative results can- 
not be drawn without appropriate caveats and speculative 
corrections. Detailed investigation of high redshift galaxies 
without significant rest-UV flux will most likely require a fu- 
ture generation of telescopes and instruments capable of per- 
forming rest-frame optical spectroscopy on extremely faint 
sources. 

The importance of this population is difficult to quan- 
tify. If the most massive sources collapsed earliest then old 
galaxies could conceivably be amongst the brightest high 
redshift sources and hence retain a measurable ultraviolet 
flux to any given magnitude limit despite the rapid decline 
of ultraviolet emission after its initial peak. Many old galax- 
ies at 2: > 5 may also have ongoing star formation and so 
examples have been identifled as high redshift sources on the 
basis of recent starburst activities rather th an through their 
evolved populations (e.g. lEvles et"al ] l2005l ). The population 
of old galaxies not-selectable as V- or 7?-drops may be con- 
strained through measurements of star formation at 2 > 7, 
but before that is necessarily a matter of speculation. 

In addition to incompleteness in terms of old, mas- 
sive galaxies, Lyman-break samples may also be incom- 
plete for you ng galaxies of similar mass to those observed. 
IVerma et al.l (2007) fitted the optical and ultraviolet spec- 
tral energy distributions of 2 ~ 5 D-band dropouts, and de- 
termined that the typical age of such galaxies is 30 Myr, 
with approximately one third of the sample showing evi- 
dence for underlying older (> 100 Myr) starbursts. While 
there is clearly a bias against old galaxies when selecting 
on unobscured ultraviolet luminosity, an intermediate age 
population of 10 — 100 Myr st arbursts should have been eas- 
ily detected. As IVerma et al.l discuss the presence of such a 
large population of short-lived sources distributed through- 



out the comparatively long (325 Myr) time span probed by 
the sample suggests that the detected galaxies represent a 
more passive population an order of magnitude more numer- 
ous that shows stochastic bursts of star formation. 

Neither of the above selection effects accounts for the 
additional effect of dust extinction, which can suppress the 
rest-frame ultraviolet flux. At lower redshifts (2 = 1 — 4) sev- 
eral species of dust-obscured galaxies are known including 
sub- millimetre galaxies and ultra-luminous infrared galaxies 
(ULIR GS). The generally blue rest-frame UV slope at 2 > 
5 (e.g. IStanwav. McMahon. fc Bunkeij|2005l : iBouwens et al.1 
200d) suggests that dust extinction at high redshifts may be 
lower than those observed at 2 « 3. A dust e xtinction curve 
deriv ed from observations of 2 > 6 quasars (|Maiolino et al.l 
l2004h suggests that high-redshift dust is generated primarily 
in supernovae and produces up to a magnitude less extinc- 
tion in the ultraviolet than the more-processed dust seen at 
lower redshift. The same extinction curve provides a good fit 
to the spectrum of a gamma ray burst that also lies at 2 > 6 
(|Stratta et al.l [20071 ). If these results are typical of galaxies 
at high redshifts, then the population lost to dust extinc- 
tion down to any given magnitude is likely to be smaller 
than that at lower redshifts, although the presence of sub- 
millimetre galaxy analogues cannot be ruled out, and mildly 
dust-obscured dropouts could be a significant cause of in- 
completeness at the faint end of any survey. 

Sample incompleteness over the sensitive redshift range 
of Lyman-break samples is dominated by the unobscured 
rest-frame ultraviolet flux and the strength of spectral 
breaks, and hence are less filter-dependent than those dis- 
cussed above. These effects are, however, significantly harder 
to quantify since they rely on the existence of a population 
that has never actually been observed. 



3 CONTAMINATION ISSUES IN DROPOUT 
SAMPLES 

Two distinct populations of astronomical objects - interme- 
diate redshift elliptical galaxies and cool Galactic stars - are 
degenerate in colour with high redshift galaxies. 

While many such contaminants can be distinguished 
through the use of data in the Spitzer/YRAC bands long- 
wards of 3 /^m, such imaging is not always available, and in 
many cases is impossible to obtain because of source confu- 
sion at long wavelengths. Similarly, a spectroscopically com- 
plete Lyman-break survey can accurately correct for con- 
tamination effects, but this realistically limits such a survey 
to limits of / = 26.5 or brighter on an 8m telescope. 

Hence understanding and correcting for these contam- 
inant populations is vital when interpreting a photometri- 
cally selected sample, particularly at faint magnitudes. 



3.1 Cool Galactic Stars 

Cool galactic stars of classes M4 and later are routinely se- 
lected in dropout selections. M class stars satisfy V or R- 
drop colours, while L and T stars have /-drop colours (figure 

Ell- 
in fields with HST/ ACS imaging, stars are routinely 
excluded on the basis of their unresolved full-width half- 
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maxim4f|. However this approach cannot be used in wide- 
field imaging, since all known (unlensed) 2 > 5 galaxies 
are unresolved from the ground. Instead the us ual approach 
adopt ed (e.g. the Subaru Deep Field studies, lOuchi et al.l 
l2004al ) attempts to exclude stars on the basis of detected 
flux shortwards of the nominal Lyman-a break or through 
placing constraints on a second colour such as I ~ Z. 

As figures [S] and [5] illustrate, the reliability of such a 
technique depends both on the depth of the imaging avail- 
able and the metallicity of the stellar population. 

In figure [8] the spectra of M and L class stars (con- 
structed from a relati vely bright sample in the Sloan Digital 
Sky Survey or SDSS, iHawlev et a"l]|2002l ) are convolved with 
the fiher profiles of HST/ ACS and those used by VLT and 
Subaru surveys. While the B ~ I colours of cool galactic 
stars are less extreme than the technically infinite colour 
of high redshift galaxies in these bands, they can nonethe- 
less reach very large colour decrements. To reliably exclude 
faint stars from a survey reaching z = 5.5 would require 
B-band imaging some five magnitudes deeper than the sur- 
vey 7-band limit. For example, given the i' = 26.5 limit o f 
the Subaru Deep Field Riz selection of lOuchi" et al.l(|2004al ). 
B band imaging to B = 31.5 would be required to reliably 
eliminate the majority of late M and early L class stars from 
the selection. The actual limiting depth of the Subaru Deep 
Field B = 27.8 is insufficient to reliably exclude stars with 
_R-drop colours. However, there is a small redshift region 
around z = 5 in most filter sets which should be clear of 
stellar contamination if the SDSS stellar templates are rep- 
rese ntative of stars at faint er magnitudes. 

'Stanwav et"ai] l|2007bl ') discuss the properties of a sam- 
ple of M-stars selected as unresolved u-band dropouts {vaao — 
i'775 > 1.3) using deep HST/ ACS imaging of the Great Ob- 
serva tories Origins Deep Survey (GOODS, iGiavalisco et al.l 
|2004| '1 reaching a limit of i' — 25. The optical and infrared 
(and, for a subsample, spectroscopic) properties of these un- 
resolved sources were studied and found to be consistent 
with those of stars and inconsistent with those of high red- 
shift galaxies. Stars at the faint magnitudes probed by high 
redshift surveys are likely to lie at large heliocentric dis- 
tances and well out of the plane of the galactic disk. Thus a 
halo origin and sub-solar metallicities is likely to be a rea- 
sonable model for such stars. Spectroscopic results for faint 
M-stars are consisted with slightly sub-solar metallicities, 
and a metallicity spread between solar an d a tenth solar 
(both photometrically and spectroscopicallv. IStanwav et al.l 
a range which can change the colours of late M stars 
by 0.4 magnitudes in both R — I and I — Z (figure[9l) . Impor- 
tantly, at early M subtypes, the effect of sub-solar metallicity 
is to produce bluer I — Z colours, and a unresolved sources in 
the GOODS fields were found to lie >0.1 magnitudes blue- 
wards of the stellar locus expected for SDSS stars with sim- 
ilar V — i' colours. Hence even a cut in / — Z is not robust 
against faint M stars. 

Combining these two effects, it is difficult or impos- 
sible to reliably eliminate stellar contamination in _R-drop 
samples given optical imaging alone. The scale of this prob- 



^ While this risks omitting compact galaxies, all confirmed z > 5 
galaxies thus far observed from space are resolved. 



lem for ground-based imaging is discussed in lStanwav et al] 
(|2007bl ). 

In ground-based surveys distinguishing stars from high 
redshift galaxies on the basis of their compact morphology is 
not possible. We quantify the effects of stellar contamination 
in such seeing-limited imaging surveys using the selection 
criteria of the Subaru Deep Field as an example. W e use th e 
n-drop selected GOODS sample of IStanwav et all (|2007bh . 
transforming the bvi' z' photometry from that measured in 
the HST/ ACS filter set to the Subaru/Supri meCam filters, 
using the convolution of stellar templates from lHawlev et al] 
(2002) with the appropriate instrument response to define 
the transformations as a function of i'—z' colour. The colour- 
magnitude distribution measured at high signal-to-noise in 
the deep GOODS imaging is then bootstrap resampled to 
define a population of ten thousand sources and their mag- 
nitudes and colours perturbed by photometric errors as a 
function of magnitude calculate d from the limiting depth 
reported bv lOuchi et al.l l|2004ar ) in each band. 

If the average stellar population of the two GOODS 
fields is representative of high galactic latitudes and large 
heliocentric distances more generally, the fraction of v- 
drop selected stars that would simultaneousl y satisfy the 
R — I, I — Z and B-band selection criteria of lOuchi et al.l 
(|2004al ) is some 5% of the underlying cool stellar popula- 
tion. Given th e surface density of v-drop stars observed by 
IStanwav et al.l (|2007bl ). this equates to an estimated con- 
tamination o f 34±1 2 stars satisfying the Riz selection of 
lOuchi et all (|2004al ) in the SDF and SXDS fields (in total 
1290 arcmin^). The large uncertainty in the stellar contami- 
nant contribution arises primarily from the field-to-field vari- 
ation in surface density of M and early-L class stars (counts 
vary by 34% between the GOODS-N and GOODS-S fields). 

This calculation is inevitably dependent on model as- 
sumptions, notably that the colour transformations appro- 
priate for bright SDSS stars are suitable for those at faint 
magnitudes, and that the numbercounts of faint stars at 
Iab = 25 — 26 are similar to those at Iab = 24 — 25 (which 
show no sign of turning over) . If the numbercounts of M-class 
stars drop sharply beyond 25th magnitude, the fraction of 
the underlying stellar population satisfying the Riz criteria 
will drop from 5% to 3%. Clearly, larger studies of faint cool 
stars in archival HST imaging is desirable to better constrain 
the behaviour of the population at these faint magnitudes. 
However, even with these caveats, we estimate that a sub- 
sta ntial fraction (20-30%) of the 106 Riz sources reported 
by lOuchi et al.l are potentially faint stars that cannot be 
identified through ground-based optical imaging alone. 

Fortunately, the susceptibility of _R-drop samples to 
stellar contamination is less severe in different filter sets, 
and will also depend on the existence and depth of aux- 
iliary imaging. The distribution of stars between M class 
subtypes creates a clear stellar locus, decreasing in number 
density with increasing subclass/redder colours. It is possi- 
ble, therefore to attempt to mask the small area correspond- 
ing to the stellar locus in a selection, either along its length 
(thus excluding any galaxies underlying it) or only where 
it doesn't cross the galaxy locus (thus including stars with 
identical colours to the galaxies concerned). Clearly in either 
case, the ability to perform this separation relies on accu- 
rate photometry and the redshift/sub-class at which the loci 
overlap. 
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Figure 9. Th e R — I and I — Z colours of model dwarf star atmo- 
spheres (from [Xllard fc Hauschildlj|l995l . , triangles), convolved 
with the Subaru/SuprimeCam filter set. Points at different metal- 
licity for the same model temperature are joined by lines and the 
metallicity in terms of [Fe/H] is marked. Metallicity tracks are for 
T=2000K (solid), 2500K (dotted), 3000K (dashed) and 3500K 
(dot- dash) . The colours of local cool stars from tHa wlev et al.l 
l2002h are shown as filled circles. The dark and pa le shaded re- 
gions indica t e the s election criteria of lYoshida et al.i (i2006.) and 
lOuchi et al.1 lHooiJ) respectively. 



In the VLT/FORS2 filter set, for example, the stellar 
locus crosses the basic galaxy track at both higher redshift 
and later spectral class than in the Subaru filter set (M5±l 
rather than M3±l, and z ~ 5.5 rather than z ~ 5.1). Since 
there are many fewer late M stars than early M stars, there 
will be relatively few contaminants, and there are also fewer 
galaxies at high redshift than lower redshift leading to a 
reduced contribution to the total (galaxy^-interloper) num- 
bercounts, although the late M stars that are present may 
be hard to remove with B band imaging due to their ex- 
treme colours. 1^-drop samples are also less prone to stellar 
contamination (lying further from the stellar locus) and will 
only detect the most metal-poor mid-M stars, or possibly 
very late M and early L stars at the high redshift end of a 
sample. 

The use of a infrared colours can also help to distinguish 
galactic stars from a z ~ 5 sample or other contaminants, 
although, as figure [10] illustrates this approach becomes less 
useful with higher redshift samples. Ideally the combination 
of data in both the near-infrared and the 3.6 /xm band of 
the Spitzer Space Telescope IRAC instrument provides the 
cleanest separation of the dropout categories, since mid to 
late M stars can have comparable colours to high redshift 
galaxies in I — K. This, of course, presents its own difficulties 
for deep surveys due to the large point spread function and 
relatively shallow confusion limit of Spitzer when compared 
with optical imaging. 



3.2 Intermediate Redshift Galaxies 

The second major source of contamination in dropout sam- 
ples - and the major contaminant from space or at faint 
magnitudes - is from low luminosity galaxies at intermedi- 
ate redshifts. Such galaxies fall into two categories: dusty, 
starforming galaxies or old, red ellipticals. 

Intense starbursts can lead to rapid production of large 
quantities of dust. In the presence of dust, the rest-frame ul- 
traviolet light is suppressed and re-emitted in the infrared. 
This results in extreme photometric colours that can imi- 
tate the Lyman break. Submillimetre observations of dis- 
tant red galaxies at z > 2 indicate that a large fraction of 
such sources have strong starb ursts, with an averag e star 
formation rate of 127M0yr"^ (|Knudsen et al.ll2005l ). The 
fraction of such sources (or their analogues across a range 
of redshifts) represented in a V- or i?-band dropout survey 
is unclear, particularly since sources across a range of red- 
shifts and with different reddening could contribute. How- 
ever, spectroscopic surveys of 7i-drops have not reported 
large contamination from emission line galaxies at lower red- 
shifts. Inter estingly, spectro scopic follow-up to narrow band 
surveys (e.g. lHu et al.l2004 ) has found low redshift line emit- 
ters, suggesting that this is a potential contaminant popu- 
lation that should be treated with caution. Infrared data 
should identify the majority of such sources, which will be 
bright at long wavelengths. 

Contamination by old, red galaxies at intermediate red- 
shifts is more straightforward to quantify. Strong spectral 
features at longer wavelengths than Lyman-a, primarily the 
4000A Balmer break (but also strong absorption features 
in the blueward band), produce very red colours in the 
dropout-selection filters. As figure[2]illustrates, such galaxies 
can easily satisfy a V- or _R-drop colour selection, and with 
the addition of intrinsic variation in colour due to varied 
star formation histories can also enter an /-drop selection 
for 2 ~ 6 galaxies. 

Although the term Extremely Red Object (or ERO) 
is often used as a short-hand for such sources (e.g. 
IStanwav et al.l |2004| ). the contaminant population for R- 
drop samples can be less extreme in R — K or I — K colour 
than conventiona l ERQs (defined as h aving / — if > 4 or 
R - K > 5, e.g. ISimpson et~aLl 120061 ). As a resuh, V ~ I 
ox R ~ I dropout samples select sources preferentially at 
0.5 < z < 1.0 and 0.6 < z < 1.6 respectively (as compared 
to the ERO population which has < z >~ 1.5). 

In most galaxy surveys, this redshift range is surveyed 
primarily with the use of photometric redshifts. While these 
are accurate for the majority of sources, they suffer from in- 
creased risk of catastrophic failure for atypical galaxies such 
as high redshift or very red sources, simply because the prob- 
ability of obtaining a unique redshift solution decreases when 
the colours of more than one population become degenerate, 
or if rare galaxy types are not represented in the spectro- 
scopically confirmed or modelled galaxy template set. 

Some high redshift samples have exploited these large 
photometric redshift catalogues and simulated predicted 
contamination from galaxies alone yielding esti mates as high 
as 40% contamination for the Riz sample of lOuchi et al.l 
(|2004bl ) (and 26% for their Viz sample, in both cases not 
taking into account the surface density of stars). Such statis- 
tical simulations of contamination are reasonable, although 



Selection Issues in z ^ 5 Surveys 11 




Figure 8. The R — I (a, left) and B — I (h, right) versus I — Z colours of observed dwarf star templates l lHawlev et aljliooj ). convolved 
with three commonly-used filter sets. In the left hand panel the redshift tracks of a fiat spectrum galaxy are also shown. The depth of 
_B-band imaging required to eliminate dropout stars can vary by more than half a magnitude depending on filter profile, and exceeds 
that available in most deep surveys. 




Figure 10. The optical-infrared colours of cool stars (diamonds), low redshift elliptical galaxies (dot-dot-dash ^ar aston!^2005!, see section 
I3.2| l and high redshift galaxies modelled as being flat in /i/ (solid), and with ages lOMyr (dotted ), 50Myr (da s hed) and 100 Myr (dot- 
dash). Optical to near-infrared colours were calculated using the observed spectra of cool stars from lKnapp et al. I ll2004l 'l. while K — 3.6 fim 
colours were derived from the results of lPatten et al.l l|2006l ) Filters plotted are the VLT/FORS2 R and 7-band, the widely-used Mauna 
Kea Kg band and the 3.6 ^m band of Spitzer/IRAC . The three populations are well-separated at z = 4 — 5 but the distinction begins 
to blur around z = 6. 



they require a detailed understanding of the selection func- 
tion in a given survey and thus cannot be applied more gen- 
erally. 

However, sp ectroscopic surveys such as DEEP2 
l|Davis et al.ll2003l fl have now characterised the population 
around z — 1 directly, measuring precise redshifts as well as 
B, R and I photometry from the CFHT. As figure [11] illus- 



trates, their sample include s galaxies that will ea sily satisfy 
a cut based on R~ I colour. IWillmer et al.l (|2006l ) examined 
the DEEP2 galaxy luminosity function at 0.7 < z < 1.4, di- 
viding their spectroscopic sample by i? — / colour as well as 
R band magnitude. Their red galaxy sample, which is a good 
match for _R-drop samples at Subaru and the VLlQ, provides 
a nearly-complete spectroscopic analysis of the dropout in- 



* http://deep.berkeley.edu/ - we use Data Release Two 



5 An i?, - 7 cut of 1.2 in the Vega system and with the CFH12K 
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Figure 11. The R — I colour distribution of potential contami- 
nant galaxies at intermediate redshifts. The colours of a mature 
elliptical galaxy (formed a.t Zf =5) from the population synthesis 
models of Maraston ( 20051) are shown for the R and / filters of the 
CFH12K instrument used for the DEEP2 survey. Points indicate 
the colours of spectroscopically confirmed 0.7 < z < 1.2 DEEP2 
galaxies with R — I > 1.0. 



termediate redshift galaxies with 18.5 < Rab < 24.1. Using 
these data, the authors were able to fit Schecter function fits 
to the luminosity function extending below the knee of the 
function in redshift bins at 0.6 < z < 0.8, and 0.8 < z < 1.0 
and to just above the knee at 1.0 < z < 1.2. In doing so, they 
find a typical volume density cj)*= 1.35 x 10"'^ galMpc"^ at 
Alg — —21.0 and z = 0.9, with a relatively shallow faint end 
slope a = 0.5 with only 'modest' redshift evolution. 

In figure [12] we calculate the predicted number density 
o f sources fo r a pop ulation with this luminosity function and 
a lMarastonI l|2005l ) spectral energy distribution suitable for 
a low redshift evolved galaxy. We use as our templat e the 
stellar population synthesis models of iMarastonI |200^ and 
cons ider a composite stellar population fo rming at z = 5 
(e.g. iThomas et al]l2005l : iLabbe et al]|2005l ') and with a star 
formation rate that decays exponentially on a timescale of 
0.5 Gyr. As figure[Tl]illustrates, this provides a reasonable fit 
to the properties of observed intermediate redshift galaxies 
with dropout colours. We note that there is an inevitable 
scatter in colour of i?-band dropouts at intermediate red- 
shifts due to the variety of star formation histories. This 
scatter is both bluewards and redwards of our fiducial model, 
but biased towards the blue. As a result, the effect of pho- 
tometric error will be to scatter more intermediate redshift 
galaxies into a colour selection than out of it. We thus re- 
gard this model as a reasonable, if not slightly conservative, 
approximation to the colour of this population. 

Peaks in the redshift distribution of interlopers can be 
seen when the 4000A break enters the I band and also at 
redshifts where a combination of emission features (such as 
[O II] A 3727A) in the redward band and absorption in the 



instrument corresponds closely to the same colour cut measured 
in AB magnitudes in the Subaru/SuprimeCam filters 
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Figure 12. The redshift distributions of intermediate redshift 
dropout galaxies, selected using the same colour criteria as given 
in table [T] and used in figure |4] The intermediate redshifts are 
modelled as elliptical galaxies with a formation redshift Zf = 5 
and number counts are generated to a limit of Ia_b = 26, based 
on the luminosity functi on derived for spectr oscopically confirmed 
red galaxies at z = 0.9 dWillmer et al.ll2006l. see text). A typical 
scatter of ±0.15 magnitudes is assumed, due to scatter in both 
the intrinsic colours and the photometry at these faint limits. 



blueward band (such as the calcium and magnesium fea- 
tures) boosts the dropout colour. 

However figure [12] does not incorporate constraints on 
the colours of these sources bluewards of the break. As figure 
[T2]illustrates, evolved galaxies (63% of the sample) can have 
colours of B — 7 > 4 at 0.5 < z < 1.2 in all the filter sets 
discussed here. Hence, the majority of contaminant galaxies 
can only be removed with confidence at bright magnitudes 
(J < 24 in the Subaru Deep Field, for example), while a sub- 
stantial population of faint contaminants is likely to remain. 

Assuming that the luminosity function and B — I distri- 
bution discussed above are appropriate approximately one 
magnitude deeper than probed by the DEEP2 survey, it is 
possible to estimate the contamination from intermediate 
redshift galaxies that satisfy all the colour constraints of 
the Subaru Deep Field survey (including 1 o non-detection 
in the B band). Given these constraints, a surface density 
of 0.05 gal arcmin"^, or a total Riz interloper count of 64 
galaxies (in a tot al of 106 Riz sources ) would be expected 
in the analysis of lOuchi et al.l (|2004al ). The surface densi- 
ties of intermediate redshift interlopers are comparable to 
those expected for 2 ~ 5 galaxies (e.g. 0.11 ga l arcmin"^ to 



I = 26 in the GQQDS-S, iBremer et al]|2004l ). and broadly 
consistent wi th the 40% low red shift galaxy contamination 
estimated bv lOuchi et al.l l|2004al ) from photometric redshift 
catalogues and simulations. 

As figures [4] and 1 121 make clear, differences in filter pro- 
file can have significant effects on the contaminant distribu- 
tion and fraction from intermediate redshift galaxies. 

As regards the V band selections, the Subaru Viz se- 
lection is cleaner than that of the HST/ ACS in large part 
due to avoiding the interloper galaxy tracks with its colour 
criteria. However this is not the sole reason. It is also critical 
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that the V — I colours of high redshift galaxies in the Sub- 
aru bands continue to increase rapidly ai z > 5 while those 
measured using ACS turn over (see figure [2} . As a result 
there is never as much separation between the galaxy loci 
at high and low redshift. It follows that even a two-colour 
selection in the HST/ ACS colours cannot be as clean as that 
in Subaru given scatter in the low redshift galaxy locus. 

Similarly, although the R-drop selection of VLT/F0RS2 
is a single colour criterion, that colour limit never includes 
any part of the low redshift galaxy track, but does include 
all the high redshift galaxies at z > 5, essentially without 
an upper redshift limit (or rather with one set by limiting 
magnitude rather than colour). The resultant ratio of high 
redshift galaxies to contaminants is high, and the absolute 
number of contaminants low. 

By contrast the single colour criterion of CFHT incor- 
porates brief redshift regions between around z — 0.8 and 
z = 1.3 for which every low redshift galaxy (and some scat- 
tering in from outside that redshift regime) have identical 
colours to high redshift galaxies and will be selected. Hence 
the number of contaminants is high and the ratio of high z 
galaxies to contaminants is low. 

Finally, the Subaru Riz selection does theoretically ex- 
clude low redshift galaxies, but the diagonal constraint in 
the colour-colour plane is within the range of photometric 
scatter and variation in the intrinsic colour of interlopers 
across a wide redshift range. In addition, the i' — z' con- 
straint applied to the Subaru selection limits the number of 
2 > 5 galaxies entering the selection window, as well as the 
low redshift population. Through a combination of these ef- 
fects, the Subaru R sample has a relatively low ratio of high 
to low redshift galaxies selected. 

In every case, the qualitative discussion above is for 
our fiducial model. We note that scatter in the colour of 
intermediate redshift galaxies to the red of our model will 
inevitably lead to more interloper galaxies entering dropout 
samples, particularly in the case of single colour selections. 
We also note the importance of near-infrared imaging where 
available. As in the case of stars, the majority of interloping 
galaxies can be separated from a high redshift sample using 
optical- infrared colours (see figure llOp . However, given the 
scatter in star formation histories and hence infrared colours 
contributing to this population, such a separation is unlikely 
to be clean. 



4 INTERPRETATION OF SELECTION 
FUNCTIONS 

4.1 Optimising a Dropout selection 

In sections [2] and |3] we discussed both completeness and con- 
tamination efi^ects that apply to z ^ 5 surveys. In many 
cases the effects compete, with any attempt to increase the 
completeness of a sample also increasing its susceptibility to 
contamination by lower redshift sources. 

This confiict between reliability and completeness is 
well understood in radio astronomy, where it applies to the 
difficult challenge of matching radio sources with the ir op- 
tical counterparts (|Condon. Balonek. fc Jauncevlll975h . Ra- 
dio astronomers define a likelihood distribution based on 
the surface density of both radio galaxies and faint op- 



tical sources to determine the optimum combination be- 
tween maximising completeness and minimising the number 
of false matches. 

The challenge for high redshift samples is less well de- 
fined, although the discussion in section |3] above demon- 
strates that it is now possible to characterise the surface 
density of contaminant sources, at least at z ~ 5. Taking 
into account all of the above constraints is clearly essential 
when comparing samples collected with a disparate collec- 
tion of instruments and selection criteria, as discussed in 
section 14.21 below. However, it may be possible to account 
for them in the early stages of a project design. In short, is 
it possible to optimise the filter combination and depth of a 
survey? And by what criteria should the 'best' selection be 
judged? 

As figures |3] and |4] illustrate, a clean-edged redshift dis- 
tribution is best attained using a selection colour that varies 
smoothly with redshift. This has the added advantage of al- 
lowing a crude photometric redshift to be determined based 
on colour alone for continuum sources (although not for 
emission-line galaxies). Avoiding plateaus and discontinu- 
ities in the colour requires filter sets in which the R and 
I bands neither overlap to a significant degree nor leave an 
unprobed redshift region between them. The ideal of square- 
sided filter response curves, abutting one another in wave- 
length, would provide the smoothest variation in colour with 
redshift but is unattainable given the limitations of interfer- 
ence filters. 

Even in this ideal, no magnitude-limited sample is going 
to present a constant selection function across the R- or V- 
drop redshift range. The effects of the galaxy luminosity 
function (which is still poorly known from spectroscopically 
confirmed sources at z > 5) must be taken into account when 
calculating the intrinsic properties of any resulting sample. 

Simultaneously minimising the filter overlap and sep- 
aration has important consequences too for the equivalent 
width-dependent selection function of a survey. No survey 
based on a simple dropout criterion is going to be simultane- 
ously complete for Lyman-a emitting and absorbing galax- 
ies over a given redshift range, without also including many 
galaxies lying outside that range. The simpler the selection 
function in terms of Lyman-a redshift, the more easily this 
important completeness issue can be modelled. 

Surveys aiming for easy photometric foUowup may wish 
to prioritise sources lying well away from the main high- 
redshift galaxy locus in order to secure detection of Lyman- 
Q. It follows that a spectroscopic survey in which all con- 
firmed sources are either very blue in I — Z or very red in 
R — I is likely to be significantly incomplete of Lyman-break 
sources at the same magnitude limit. 

Of course, such a survey must also account for the other 
half of the completeness versus reliability dilemma. 

The high surface density of contaminant sources dis- 
cussed in sections 13.11 and 13.21 highlights the importance of 
modelling contamination in any given filter set not only from 
intermediate redshift galaxies but also from extreme Galac- 
tic stars. This constraint is particularly important for large 
area surveys observed from the ground since their shallower 
limits and large survey area leaves them highly vulnerable 
to stellar contamination. 

As shown in figures |8] and 1131 deep imaging in bands 
shortwards of the Lyman-limit at z « 5 can help to elimi- 
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Figure 13. The B-band depth required to eliminate contami- 
nant galaxies at intermediate redshifts. The colours of a mature 
elliptical galaxy (formed at Zf=b) from the population synthesis 
models of iMarastoiil l|2005h are shown for the B and / filters of 
the Subaru dataset and for the CFH12K used for the DEEP2 sur- 
vey (see text). Spectroscopically confirmed 0.7 < z < 1.2 galaxies 
satisfying (R — I) ab > 1-4 from the DEEP2 survey are shown for 
comparison. 



nate a large fraction, but by no means all, of the contam- 
inants. Surveys aiming to eliminate contaminants based on 
optical photometry alone must necessarily reach exceptional 
depths in the bluewards band. Even then, given the high in- 
trinsic scatter in the colours of contaminant populations, the 
effects of the contaminants on any derived results must be 
calculated for the appropriate survey (as discussed in section 

A second line of defence against contaminants may be 
obtained from the use of near-infrared imaging. Those stel- 
lar contaminants with the most extreme B ~ I colours (i.e. 
very late M and L stars) are also those most easily detected 
in near-infrared, and thus the blueward and redward bands 
are complimentary in removing contaminants from the high- 
redshift dropout sample. Optical-infrared colours may assist 
in identifying contaminating galaxies at intermediate red- 
shifts. However, if the scatter m B — I colour is typical of 
the range of star formation histories contributing to an in- 
termediate redshift dropout sample, then a similar scatter 
might be expected longwards of the drop colours and hence 
the use of near-infrared filters cannot guarantee a clean sam- 
ple. 

A 2: > 5 survey intended to obtain the maximum com- 
pleteness and minimum contamination from lower redshift 
sources requires imaging across a broad wavelength range, 
incorporating not only the dropout colour, but also depth- 
tuned imaging both to the blue and in the infrared. Given 
that dropout populations - both at high and low redshift 
- comprise sources with non-smooth SEDs, the colours are 
filter-dependent and their properties in any survey must be 
carefully calculated to determine the appropriate matched 
depths. Even then no Lyman-break survey will ever be 



complete for non-starforming, passively evolving galaxies at 
high-redshifts. 

While noting all the points above, in figures [T4l and [161 
we illustrate in basic terms the completeness and contami- 
nation of samples derived from the selection criteria in table 
[1] Figure [14] shows the fraction of all galaxies (integrated 
to infinite faintness) detectable as a function of limiting 
magnitude at three different redshifts and for each selec- 
tion function. As in section [2] we u se the luminosity f unc- 
tion of 2 = 3 Lyman break galaxies (|Steidel et al.|[l999l ) for 
reference, while noting that changes to the shape of the lu- 
minosity function make little difference to the comparative 
behaviour of the selection functions (as discussed in detail in 
section [2TT} , but have a rather larger effect on their normal- 
isation. As a result, the numerical values in figure [T4] should 
be viewed as indicative rather than accurate. 

In figure [TS] we consider a slightly different parameter, 
showing instead the fraction of galaxies with a continuum 
magnitude measured at 1500A (rest) of misoo = 26.0, re- 
covered by Monte Carlo simulations to a detection limit of 
/ = 26.0 as measured in each filter set. Model galaxies were 
distributed in continuum magnitude according to the mea- 
sured z — 3 luminosity function and their /-band magnitude 
and I — Z colour determined as a function of redshift and 
filter sets, assuming a flat rest-UV continuum. Colours and 
magnitudes were then perturbed by random photometric er- 
rors, assuming a typical error of 0.1 mag at the selection limit 
of / = 26, and the fraction of galaxies satisfying the selection 
criteria determined. Since each / band filter is suppressed by 
IGM absorption to a different degree as a function of red- 
shift, the measured /-band value can differ from the contin- 
uum magnitude by several tenths of a magnitude even for 
a fiat spectrum source. As a result continuum sources close 
to the faint selection limit can be lost when measured in 
the /-band. In some cases sources below that limit can be 
pulled up into the selection, leading to contamination of the 
sample by galaxies lying at z 5 but not strictly meeting 
a continuum selection criterion. Comparison between sur- 
veys complete to any particular magnitude is only possible 
if that limit is defined by flux in a band unaffected by IGM 
absorption. 

Hence figures [14] and [15] represent a comparison of com- 
pleteness for continuum-magnitude limited surveys against 
an ideal (infinite depth) survey, illustrating the effect of both 
redshift distribution and limiting magnitude on the recovery 
of high redshift sources. In both cases, the relative behaviour 
of the different filter combinations is similar. Only the two 
V-drop samples select galaxies at 2 = 4.6 since galaxies 
at these redshifts are too blue to be /i-band dropouts. By 
contrast all five selection functions are, in theory, sensitive 
to galaxies with a flat continuum at 2 = 5.1 and 2 = 5.6. 
At both re dshifts, the VLT/FO RS2 fllter set and selection 
function of jPouglafi et all l|2007l fl recovers a higher fraction 
of the galaxies posited by any reasonable luminosity func- 
tion than do the other filter sets discussed here, with the 
CFHT/MegaCam standard filter responses and the Subaru 
/?-drop selection performing least well. 



^ Note: iLehnert 
their R — I > 1.5 cut reduces completeness at 



Bremei] bOO^ ) also used this filter set but 

: 5.1 while 
leaving it unaffected at z = 5.6. 
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Figure 14. The fraction of galaxies recovered by different survey 
selections as a function of redshift and limiting magnitude. The 
total number of galaxies at a given redshift is predicted in each 
case by integrating the luminosity function to infinity. The lumi- 
nosity functio n appropriate to Lyman break galaxies at 2 = 3 
llSteidel et al ] 11999) 

is applied here at all redshifts and hence 
fractions should be considered indicative rather than accurate. 
Changing the luminosity function alters the normalisation of this 
plot with little effect on the shape or relative response of different 
surveys (see text). Line colours and styles are as in figure[3] Frac- 
tions at different redshifts are shown with different line thickness 
and symbols. 



In figure [16] we illustrate the other side of the equation, 
using the lumino sity funct i on det ermined for red galaxies at 
z — 0.9 by Will mer et al.l l|2006l ) to predict their selection 
efficiency by any given filter and instrument combination. In 
each case only selection on RoxV ,1 and Z is assumed since, 
as demonstrated above, any associated B-band is usually 
too shallow to reliably eliminate a sizable fraction of these 
sources. 

Again the VLT/FORS2 filter set performs weU with low 
sensitivity to contaminants in both redshift ranges 0.4 < z ^ 
1.0 and 1.0 < 2 ^ 1.6. Overall, the V- band selection func - 
tion apphed at Subaru/SuprimeCam bv lOuchi et al.l (|2004al ) 
suffers least contamination being completely insensitive to 
contaminants at 0.4 < z ^ 1.0 and only weakly sensitive to 
them at 1.0 < z < 1.6. Although the broad HST/ACS y and 
6-bands used by the GOODS survey (|Giavafisco et al.|[2003 ') 
render it more vulnerable to contaminants at intermediate 
redshifts, these are also easier to identify morphologically 
from space. 

The filter combinations discussed here most vulnera- 
ble to contamination by intermediate redshift galaxies are 
the RIZ filter sets commonly used by Subaru/SuprimeCam 
and CFHT/Megacam. These samples are also those most 
susceptible to stellar contamination as discussed in section 
13.11 In both cases, the relatively short central wavelengths 
of the filters requires that selection colours are less extreme, 
and therefore admit a larger contaminant population than 
in other filter sets. 

With the increasing availability of sensitive large for- 
mat imagers, combining optical filters from multiple facil- 



Figure 15. The fraction of galaxies with "^j^gQQ^ = 26.0 recov- 
ered to / = 26 by different survey selections as a function of 
redshift. A typical photometric error of 0.1 magnitudes at / = 26 
was assumed in each case and the intrinsic colours calculated as- 
suming a flat spectrum in Line colours and styles are as in 
figure [3] 
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Figure 16. The fraction of intermediate redshift galaxy con- 
taminants satisfying the colour selection criteria shown in table 
[T] calculated from the z = 0.9 luminosit y function determine d 
for spectroscopically confirmed galaxies bv lWillmer et al ] ||2006h . 
The intermediate redshifts are modelled as elliptical galaxies with 
a formation redshift Zf = 5 (see section[32)l and integrated across 
two redshift ranges: 0.4 < z 1.0 (thick lines) and 1.0 < z sC 1.6 
(thin lines with diamonds). Line styles are as in figure l3l 



ities may yield advantages in terms of tuning the redshift 
selection and required depth in complimentary filters. 

In designing any large Lyman break galaxy survey for 
galaxies at z > 5 it is also important to note that the best 
indicator of both contamination and completeness would be 
a spectroscopic survey reaching sensitive limits and includ- 
ing a large fraction of the candidate sources. Without such 
spectroscopy, the properties of a survey can only be mod- 
elled rather than measured. 
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4.2 Interpretation of the Existing Literature 

Taking into account the detailed consequences of the con- 
tamination, selection and completeness issues above in- 
evitably has consequences for the interpretation of results 
published and discussed in the literature. As case studies 
we consider three results that have attracted attention both 
from other observers and from theorists, and put them in 
the context of the labyrinthine selection function of 2; > 5 
Lyman break galaxies. 

The blue rest-UV colours of Lyman break galaxies at 
high redshift has already been the subject of speculation 
and interpretation. While Lyman break dropout galaxies 
are largely consistent with young starbursts, their rest- 
UV slopes have generated comment are in many cases 
too blue to fit with standard population synthesis model s 
fe.g. lYan et al ] |2005l : IStanwav. McMahon. fc Bunkejl2005l l. 
Analy sing a samp le of galaxies from the Hubble Ultra Deep 
Field, lYan et ahl (|2005, ) suggested that either a top-heavy 
initial mass function (IMF) or a lower than predicted in- 
tergalactic medium absorption at high redshifts could ex- 
plain this result. While most of their galaxies have not been 
confirmed spectroscopically, at least two of the galaxies de- 
scribed as unusually blue have indepen dent spectroscopy 



iRhoads et al.ll2005l : IStanwav et all 



2004) and well-detected 



Lyman-alpha emission lines. Taking Van et af] object #15ab 
as an example, the source is known to have an emission line 
with Wo=70A at z — 5.4. Given this, and the effect on 
colour illustrated in figure [SI its colours {v — i' = 2.9 ± 0.3, 
i! ~ z = 0.24 ± 0.05) are completely consistent with those 
of a flat spectrum source. Combining such an interpretation 
with the distribution of high equivalent wid ths observed in 
spect roscopy of the Hubble Ultra Deep Field (|Stanwav et al.l 
[2003), it is not unreasonable to assume that most or all of the 
blue sources in this field have Lyman-a emission lines (with 
equivalent widths explainable by standard IMFs) pointing 
to a young popula tion with sporadic bursts of intense star 
formation (see also lVerma et al]|2007h . 

Accurate measurements of the rest-frame ultraviolet lu- 
minosity function of 2; « 5 are crucial to interpreting the 
role of the population in cosmological processes such as the 
reionisation of the universe and mass build-up of the largest 
present day galaxies (given assumptions for the IMF and 
population age). Determination of the luminosity function 
is limited by three key parameters: the area of the survey, 
the depth in comparison to the typical luminosity L* and 
the degree to which the photometric sample represents the 
underlying population. Despite these limitations, estimated 
a « 5 luminosity functions have already been u sed by the- 
orists to constrain both of these processes (e.g. iMao et al] 
I2OO7I : iNight et al.ll2006l l . 

Two separate luminosity functions have been derived 
from 2; ~ 5 photometric selections, both observed from the 
ground using the SuprimeCam instrument, bu t based on dif- 
ferent imaging filters and selection criteria. lYoshida et al.l 
l|2006l ') conducted their analysis on the Subaru Deep Field 
using the Viz and Kiz criteria described in table [T] By con- 
trast, Hw^^^eT^l] ((2007) based their survey on a VIZ sam- 
ple of a similar size, combining the Hubble Deep Field North 
and a second non-contiguous blank field, and substituted the 
7c filter for the i! used by the Subaru Deep Field team. This 
filter lies redwards of the i! and hence z « 5 sources show 



bluer I — Z colours in the llwata et al.l sample than the Sub- 
aru De ep Field sam p le, whi le having similar V ~ I colour^. 

In lOuchi et all (12004^ 1. the Subaru Deep Field team 
estimate the fraction of low redshift galaxy contaminants in 
their Viz sample as 26% based on simulations of photomet- 
ric redshift catalogues. This fraction is in good agreement 
with our estimate of the galactic contamination. Stellar con- 
tamination is not a serious problem for l/-drop surveys since 
cool stars are generally too red in 7 — Z to be confused with 
high redshift galaxies. Hence the galaxies are representa- 
tive of the total sample c ontamination. By contrast, work- 
ing with the same dataset, Yoshi da et al.l estimate their con- 
tamination as only a few percent suggesting that a degree of 
uncertainty exists concerning the properties of the sample. 

Furthermore, the Subaru Deep Field Viz sample is sup- 
plemented by the the Riz sample which comprises 30 per 
cent of the 2: « 5 total. As discussed in sections 13.11 and 
13.21 contam ination is likel y to b e a serious issue in the Riz 
selection of lYoshida et al.l (|2006l ). with known contaminant 
populations conceivably accounting for all of the detected 
sources, and certainly accounting for > 50%. Overall, as 
much as half the total 2 « 5 sample might be accounted for 
by contaminants. 

By contrast, the llwata et al.l survey may be compara- 
tively clean. A bluewards shift in the near-vertical V — I , 
I — Z colour track compared to the Subaru Deep Field fil- 
ters has the effect of increasing the separation between the 
colours of model high redshift galaxies and those of most in- 
termediate redshift interlopers. Nonetheless, some contami- 
nant fraction is likely to remain. The scatter in optical SEDs 
of intermediate redshift dropout galaxies (almost four mag- 
nitudes in i? — 7, as illustrated in figure I13|l suggests that 
a spread vciV — I colours of approximately one magnitude 
would not be unreasonable. Given that the separation be- 
tween model high and low redshift tracks is as small as 
0.3 magnitudes at V — Ic = 1.5 (and in the absence of 
shorter wavelength imaging), low redshift contamination is 
inevitable and will be biased towards the low redshift end of 
the sample which dominates the total number counts, par- 
ticularly at bright magnitudes. Given the relative redshift 
range over which the two l/-drop samples here are suscepti- 
ble to low red shift contam ination, we estimate the contam- 
ination of the llwata et al.l sample as 5-10%. 

The selection functions of the two 2 « 5 surveys, which 
theoretically probe similar redshift ranges, clearly have mas- 
sive consequences for the contamination fraction in their 
surveys. The distribution o n those contamina nts - biased 
to bright magnitudes in the llwata et al. I 1I2OO7I) surv e y and 
peaking in the fainter Riz sample of Yoshida et al.l ( 20061 ) 
may go some way to explaining the differences between their 
derived luminosity functions. The function mea sured by the 
SDF team is steeper, finding fewer galaxies than llwata et al.] 
at bright magnitudes, while finding higher surface densities 
at faint magnitudes. Contaminants are unlikely to explain all 
the difference between the two luminosity functions, but cer- 
tainly constitute a contributing factor which must be care- 



^ Note: The colour-colour plots in lYoshida et al.l fcOOfit) and 
other Subaru Deep Field papers appear to show a systematic off- 
set inV — I with respect to those of other surveys, although this 
offset is applied uniformly to data, models and selection criteria 
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fully accounted for as a function of magnitude and redshift. 
As a result, it is likely that neither the relati vely shallow 
lumin osity function slope of a = —1. 48 ± 0.3 ('iwata ct al] 
[2003) or a steeper a = -1.8 to -2.3 (|Yoshida c t al. 2 0oi ) 
accurately describes the population at z « 5. 

Although modelling is shaped by observation, rather 
than the reverse, comparisons between observed photomet- 
ric samples and numerical models support the possibility of 
higher than s upposed contaminatio n in t he Subaru Deep 
Field Sample. iKitzbichler fc White! (|2007l ) used the large 
Millennium Simulation as the test bed for models of galaxy 
formation. Although these models provided a good fit to 
galaxy number counts at bright magnitudes and moderate 
redshifts, they predicted a factor of 2-8 times fewer galaxies 
at 2: > 4.5 than observed in the SDF photometric sample. 

Since the population faint end slope is essential for cal- 
culating the ionising flux contribution of the population, the 
same contamination issue may explain the discrepancy be- 
tween the ultravi olet flux density de termined from the Sub- 
aru Deep Field (lOuchi et al.l l2004al) and other determina- 
tions at the same redshift l|Lehnert fc Bremeij|2003l ). While 
the latter, based in part on a spectroscopic survey, found 
evidence for a decline in t he ultraviole t luminosity density 
between z = 3 and z = lOuchi et al.l found that the lumi- 
nosity density shows no evidence for such a decline. Given 
this uncertainty, reionisation theorists should be wary of ac- 
cepting the results of any one survey or selection criterion 
as representative of the starforming population at these red- 
shifts. 

The third key result derived from Lyman break surveys, 
and used to constrain high redshift galaxy models and derive 
the galaxy-halo bias at z ~ 5, is on the clustering of high 
redshift dropout galaxies. 

The angular two-point correlation function w{9) de- 
scribes the overdensity of galaxies as a function of angular 
scale when compared with a randomly generated distribu- 
tion with the same geometrj{f|. This purely observational 
function yields an angular dependence /3 and clustering am- 
plitude Aw such that w^d) = Am9~^. These can be con- 
verted into a derived clustering length given assumptions 
about the redshift distribution of the sample. 

Measurements of the clustering at z ~ 5 have been 
made based on dropout galaxies in the G OODS data and in 
the Subaru Deep Field. Lee et ahl (|2006l ) have explored the 
clustering of Lyman-Break galaxies in the GOODS fields, 
using the same HST/ ACS filters but a slightly different se- 
lection function to that given in table[T] This survey is one of 
relatively few to benefit from high resolution imaging data, 
allowing sources of stellar morphology to be removed before 
analysis. The selection criteria applied in the v — i'vsi' — z 
colour-colour plane (a slight modification of the HST/ ACS 
criteria used elsewhere in this paper) should successfully re- 
move virtually all lower redshift interlopers (a fact supported 
by visual inspection of the candidates). However, it is not 
clear that the v and i' limits (just two magnitudes deeper 



* Also seen at z=6 (e.g. iBunker et al]|2004l) 
9 And is defined as «)(6»)=(GG(6»)-2GR(6»)-|-RR(9))/RR(e) where 
GO is the number of galaxy-galaxy pairs at a given separation, 
RR is the equivalent number for rando mly placed galaxies and GR 
is the number of galaxy-random pairs llLandv fc Szalavlll993f) . 
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Figure 17. The effect on clustering measurements of a highly 
clustered contaminant population. The clustering of spectroscop- 
ically confirmed 0.7 < z < 1.2 galaxies satisfying [K — I) ab > 1-4 
from the DEEP2 survey was measured, and randomly placed 
galaxies added such that the DEEP2 survey constituted between 
15% and 100% of the total sample before remeasurement. The 
resultant population is less highly clustered than that a,t z = \ 
but still shows a strong signal, even with 85% of the sample made 
up of randomly placed galaxies. T he angular cl ustering functions 
measured bylLee ct al. (200i) and lOuchi et al.l ( 2004b) for z = 5 
Lyman break galaxy samples are shown for comparison. 



than z') are sufficient to support such a clean selection at 
the faint end of the sample. To cleanly select galaxies at 
z — 5.2 (the upper end of their redshift selection) would re- 
quire a w-band limit som e three mag nitudes deeper than z' . 
Since it is not clear from iLee et al] whether non-detections 
in V are required to satisfy the v — i' criterion or merely 
be consistent with it, it is possible that the sample is seri- 
ously contaminated with low redshift galaxies in the range 
z'ab = 26- 27. 

The cost of this relatively clean selection (at least at 
bright magnitudes) is paid in allowing for relatively little 
scatter in the high redshift galaxy colours. Hence this result 
should be robust against any significant contamination from 
lower redshift sources but may suffer from incompleteness, 
particularly due to line emission towa rds the lo w redshift 
end of the selection. IStark et al.l (|20o3) assess the lLee et al.l 
samplj^ and find evidence for considerable incompleteness 
toward the low redshift end of their theoretical redshift se- 
lection window based on the omission of 12 out of 29 sources 
with spectroscopic redshifts in this range due to somewhat 
blue colours. The photometric selection completeness may 
well be better than this suggests since spectroscopic surveys 
are at present themselves biased towards the selection of line 
emitters in order to obtain a precise redshift. 

Despite this latter caveat, by excluding Galactic stars 
the analysis of clustering at z = 5 from space-based data 
may well be the most robust measurement available, given a 
sensible magnitude limit, albeit one limited to small spatial 



and the iGiavalisco et al" I ll2004h sample on which it is based 
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scales and understanding of their true redshift sensitivity 
rang e. The resu lting measurement of correlation amplitude 
from lLee et alj is slightly higher than that determined at z « 
5 from the SDF to the same magnitude limit, but consistent 
within the stated errors, and also found that the best fit 
req uired a steep angular dependence, f3 — 1.1. 

lOuchi et all (|2004bl ) determined w{e) from the SDF Viz 
and Riz samples discussed abovj^. While fixing the power 
law slope at /3 = 0.8, they found evidence for a larger clus- 
tering amplitude at z = 5 than observed at z = 4 a nd below, 
but slightly lower than that observed bv lLee et al.l . However, 
as discussed earlier, a contamination fraction of 26% is esti- 
mated by the authors, and may well be much higher if the 
colour distribution of 2; = 1 galaxies and cool stars are taken 
into account; 

In both lLee etall (|2006l ) and lOuchi eFall (|2004bl ). the 
authors make an adjustment for contaminant galaxies, but 
in doing so assume that such contaminants arise from pho- 
tometric scatter and hence are distributed randomly in red- 
shift and hence randomly across the sky (i.e. are unclus- 
tered). Stellar contaminants satisfying z = 5 dropout cri- 
teria are indeed unlikely to be signific antly correlated on 
the sky, although lStanwav et al.l l|2007br ) did find substantial 
variation from field to field. By contrast, the z ~ 1 galaxy 
population discussed in section 13.21 is known to be highly 
clustered with red g alaxies more bia sed than blue sources 
at the same redshift (|Coil et al.ll2"007l ). As a result, the clus- 
tering signal in a sample may be strengthened rather than 
weakened by the presence of contaminants. 

In figure [T7] we explore the effects of a small contam- 
inant population on the clustering signal of an otherwise 
random galaxy distribution. We use as a base the spectro- 
scopically confirmed J? — / > 1.4 dropout galaxies detected 
at z = 1 by the DEEP2 survey. These were added to a ran- 
domly placed sample distribution of galaxies such that they 
comprised a varying fraction of the total sources, and the re- 
sultant clustering function was measured. As expected, the 
clustering amplitude declines as the fraction of randomly 
placed sources increases. However, as figure [T7] illustrates, a 
contamination rate as low as 15% in an otherwise unclus- 
tered sample can produce a measurable clustering signal. 

In section 132] we calculated that approximately 25% of 
the Subaru Deep Field Viz sample is likely to lie at z « 1. 
Given the highly clustered nature of these contaminants, the 
measured angular correlation function is consistent with no 
clu stering at a ll in the target z = 5 population. Similarly 
the iLee et al] (|2006h clustering result is consistent with no 
clustering at z « 5 if just 15% of their sample lies at z = 1, 
and could have been attained within one standard deviation 
if just a few percent of their sample lies ai z — 1 (possible 
given that the numbercounts are dominated by faint source 
with non-detections in v). 

This result does not prove that the population at z = 5 
is unclustered (which would be very surprising given their 
evident placement in massive dark matter halos), but does 
highlight the fact that extreme care must be taken in disen- 



tangling the influence of highly clustered contaminant popu- 
lations contributing even a few percent to the total. Also in- 
teresting is that the DEEP2 data is too shallow to constrain 
the upturn in cluster ing seen at small sca les by z = 5 surveys 
and weU detected bv lOuchi et al.l l|2004bl ') at z = 4. A simple 
visual inspection of high redshift candidates in Hubble Space 
Telescope imaging confirms that many of them form groups 
with multiple neighbours on the scale of a few arcseconds 
or less. This small scale clustering is not seen at z « 1 and 
suggests a high galaxy-halo bias. 



4.3 Implications for Higher Redshifts 

In this paper we have focused on implications for galaxy 
surveys at z ~ 5 since this is the largest and most-widely 
studied regime beyond the more easily accessible and well- 
known Lyman-break galaxy population at z ~ 3 — 4. How- 
ever, the same effects as those discussed above are applicable 
to galaxy surveys targeted at higher redshifts. 

The effects of completeness-related issues are likely to 
become more severe at high redshift, due to the high back- 
ground in ground-based near-infrared imaging. This arises 
primarily from atmospheric OH emission in the J and H 
bands, and can vary dramatically on a timescale of hours. 
As a result, sources at certain redshifts may never be ob- 
served in Lyman-a emission from the ground due to at- 
mospheric line blanketing. The resulting broad filters, and 
gaps between filters, in the near infrared lead to less pre- 
cise redshift discrimination, and hence potentially increased 
sensitivity to filter effe cts. While the common usa ge of the 
Mauna Kea filter set (|Simons fc Tokunagal |2002| ) is likely 
to mitigate this effect, where alternate filter definitions ex- 
ist in the near-infrared they tend to differ dramatically in 
transmission profile. 

In the short term, sources identified and confirmed at 
very high redshifts are likely to represent only those sources 
with high equivalent widths in Lyman-a and hence may not 
be representative of (or, in some cases, represented in) the 
Lyman-break galaxy surveys that will follow. 

/-drop surveys, aimed at finding Lyman-break galaxies 
at z ~ 6, are susceptible to the presence of late-M , L and 
T stars as shown in figure (5] while near-infrared dropout 
samples aimed at still-higher redshift may suffer contamina- 
tion from cool, red star species such as N-type carbon st ars 
l|Hawthorn et al.ll2007l : iTotten. Irwin, fc Whiteioc5l2000D . 

Just as the redshift of Lyman-break galaxies increases 
as longer wavelength filters are used for selection, so too 
does the redshift of contaminant galaxies selected for their 
Balmer breaks. The conventional and well studied ERO 
population (e . g. Simpson et al.l 20061: Brown et al] l2005l : 



We note that iKashikawa et al] 1I2OO6I ) repeated this analy- 
sis, supplementing it with slightly deeper and simulation data 
and determining a clustering amplitude consistent with that of 
lOuchi et alj ll2004lj) . 



iGilbank et"al] l2003l : ICimatti et all 120021 ') contributes con- 
taminants to z » 6 /-drop samples, while the still-redder 
sources of lYan et al. (2004 ) or the z > 2 pop ulation dis- 
cussed by iDunlop. Cirasuolo. fc McLurd (|2007l ) contribute 
contaminants to near-infrared dropout samples at z > 6.5. 

Although the surface density of contaminant popula- 
tions decreases as the selection moves redwards, the surface 
density of target high redshift galaxies also declines sharply 
with increasing luminosity distance. Hence an analogous sit- 
uation to that at z « 5 exists at higher redshifts, and the 
issues discussed in this paper will remain crucial to inter- 
pretations of these populations. Spectroscopy to consider- 
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able depths will almost certainly be required to characterise 
the properties of any high redshift sample. Given that such 
studies form a core component of the scientific rationale for 
forthcoming instruments and facilities, a firm grasp of sub- 
tle contamination and completeness issues, and a clear re- 
porting of the methodology underlying any sample will be 
essential for some time to come. 



5 CONCLUSIONS 

The main conclusions of this paper can be summarised as 
follows; 

(i) The redshift distribution and number counts pre- 
dicted for a given survey is a sensitive function of filter 
profile, complicating comparisons of surface density between 
surveys. 

(ii) Moderate line emission is sufficient to move z > 5 
galaxies into and out of selection functions, and can lead 
to scatter from the continuum galaxy locus of more than a 
magnitude in colour for line widths of Wo = 50 A. Again, the 
scale of this effect depends sensitively on the filter profiles. 

(iii) Results derived from ultraviolet-dropout samples 
apply only to a subset of the total galaxy population. Qui- 
escent galaxies - both young and old - are likely to be missed 
by dropout samples due to faint ultraviolet continua. Dusty 
galaxies at 2: > 5, however, are less likely to be omitted 
from samples than those at z < 4 due to evolution in the 
extinction curve. 

(iv) Stellar contamination can be a serious issue for 
dropout selections based on ground-based optical surveys. 
Again this effect is filter dependent, but comparison with 
space-based data suggests that up to 30% of some pub- 
lished samples might be accounted for by stellar contami- 
nation alone. In these cases infrared data may help identify 
contaminants. 

(v) Contamination from galaxies at intermediate red- 
shifts is again sensitive to the filters used for colour selection. 
A large fraction (63%) of dropout galaxies at « « 1 can have 
B — I > 4, making them difficult or impossible to eliminate 
through optical imaging alone. The surface density of such 
extreme interlopers is of the same order of magnitude as the 
target z m 5 galaxies. 

(vi) The redshift range of a survey and its susceptibil- 
ity to contamination can both be tuned by selecting different 
filter combinations, ideally selecting square-sided transmis- 
sion profile filters. Deep imaging both blue and redwards of 
the break colours are essential to minimise contamination. 

(vii) Some contaminants, with extreme colours in all 
bands are only ever likely to be identified spectroscopically. 
The deep imaging required to do so photometrically is cur- 
rently infeasible bluewards of the break, and pushing the 
bounds of possibility in the infrared. 

(viii) Care must be taken to consider selection functions 
and contamination fractions when interpreting and compar- 
ing the results of dropout surveys. Such effects may explain 
the discrepancies between results at the same redshift based 
on different observational data. 

(ix) The clustering strength seen in z ~ 5 surveys is 
entirely consistent with that expected given a small, highly- 
clustered contaminant population in an otherwise uncorre- 
lated population. While this does not imply that z ~ 5 



galaxies are unclustered, it does cast doubt on the reliability 
of current clustering measurements. 

(x) The issues discussed in this paper will remain rele- 
vant at higher redshifts, although the populations concerned 
and appropriate surface densities evolve with redshift. 
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